Papers with testing models
Leveraging Type Descriptions for Zero-shot Named Entity Recognition and Classification (2021.acl-long)
Copied to clipboard
| Challenge: | Named entity recognition and classification (NERC) tasks require annotated data for the target classes during training. |
| Approach: | They propose a novel approach that leverages the fact that textual descriptions for many entity classes occur naturally. |
| Outcome: | The proposed approach outperforms baselines adapted from machine reading comprehension and zero-shot text classification. |
NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding (2024.findings-emnlp)
Copied to clipboard
Chunkit Chan, Cheng Jiayang, Yauwai Yim, Zheye Deng, Wei Fan, Haoran Li, Xin Liu, Hongming Zhang, Weiqi Wang, Yangqiu Song
| Challenge: | Theory of mind evaluations currently focus on testing models using machine-generated data or game settings prone to shortcuts and spurious correlations. |
| Approach: | They propose a benchmark to stress-test machine ToM in real-world negotiation surrounding covered multi-dimensional mental states. |
| Outcome: | The proposed benchmark builds upon the Belief-Desire-Intention theory and conducts the necessary empirical experiments to evaluate large language models. |
PARADE: A New Dataset for Paraphrase Identification Requiring Computer Science Domain Knowledge (2020.emnlp-main)
Copied to clipboard
| Challenge: | Paraphrase identification requires specialized domain knowledge to perform . state-of-the-art neural models and non-expert human annotators have poor performance on PARADE . |
| Approach: | They propose a benchmark dataset called PARADE for paraphrase identification that requires specialized domain knowledge. |
| Outcome: | The proposed dataset shows state-of-the-art models and non-expert human annotators have poor performance on PARADE. |
FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations for theory of mind (ToM) use passive narratives that lack interactivity. |
| Approach: | They propose a benchmark to stress-test ToM within information-asymmetric conversational contexts via question answering. |
| Outcome: | The proposed benchmark is challenging for state-of-the-art language models, which perform significantly worse than humans even with chain-of thought reasoning or fine-tuning. |